hpgltools examples using the fission dataset

1 Example hpgltool usage with a real data set (fission)

This document aims to provide further examples in how to use the hpgltools.

Note to self, the header has rmarkdown::pdf_document instead of html_document or html_vignette because it gets some bullcrap error ‘margins too large’…

1.1 Setting up

Here are the commands I invoke to get ready to play with new data, including everything required to install hpgltools, the software it uses, and the fission data.

## These first 4 lines are not needed once hpgltools is installed.
## source("http://bioconductor.org/biocLite.R")
## biocLite("devtools")
## library(devtools)
## install_github("elsayed-lab/hpgltools")
library(hpgltools)
require.auto("fission")

## [1] 0

## I use the function 'sm()' to quiet loud functions.
tt <- sm(library(fission))
tt <- sm(data(fission))

1.2 Data import

All the work I do in Dr. El-Sayed’s lab makes some pretty hard assumptions about how data is stored. As a result, to use the fission data set I will do a little bit of shenanigans to match it to the expected format. Now that I have played a little with fission, I think its format is quite nice and am likely to have my experiment class instead be a SummarizedExperiment.

## Extract the meta data from the fission dataset
meta <- as.data.frame(fission@colData)
## Make conditions and batches
meta$condition <- paste(meta$strain, meta$minute, sep=".")
meta$batch <- meta$replicate
meta$sample.id <- rownames(meta)
## Grab the count data
fission_data <- fission@assays$data$counts
## This will make an experiment superclass called 'expt' and it contains
## an ExpressionSet along with any arbitrary additional information one might want to include.
## Along the way it writes a Rdata file which is by default called 'expt.Rdata'
fission_expt <- create_expt(metadata=meta, count_dataframe=fission_data)

## Bringing together the count matrix and gene information.

1.3 Normalizing and exploring data

There are lots of toys we have learned to use to play with with raw data and explore stuff like batch effects or non-canonical distributions or skewed counts. hpgltools provides some functionality to make this process easier. The graphs shown below and many more are generated with the wrapper ‘graph_metrics()’ but that takes away the chance to explain the graphs as I generate them.

## First make a bar plot of the library sizes in the experiment.
## Notice that the colors were auto-chosen by create_expt() and they should
## be maintained throughout this process
fis_libsize <- plot_libsize(fission_expt)
fis_libsize

## Here we see that the wild type replicate 3 sample for 15 minutes has fewer non-zero genes than all its friends.
fis_nonzero <- plot_nonzero(fission_expt, labels="boring", title="nonzero vs. cpm")
fis_nonzero

1.3.1 An initial pca plot

In most cases, raw data does not cluster very well, lets see if that is also true for the fission experiment. Assuming it doesn’t, lets normalize the data using the defaults (cpm, quantile, log2) and try again.

## Something in this is causing a build loop on travis...
## Unsurprisingly, the raw data doesn't cluster well at all...
fis_rawpca <- plot_pca(fission_expt, expt_labels=fission_expt$condition)
fis_rawpca$plot

## So, normalize the data
norm_expt <- sm(normalize_expt(fission_expt, transform="log2", norm="quant", convert="cpm"))
## And try the pca again
fis_normpca <- plot_pca(norm_expt, plot_labels="normal", title="normalized pca")
fis_normpca$plot

summary(fis_normpca)

##          Length Class      Mode   
## pca       3     -none-     list   
## plot      9     gg         list   
## table     8     data.frame list   
## res       4     data.frame list   
## variance 35     -none-     numeric

In the final line of the preceeding block, I printed a summary of the return from plot_pca(). It contains the following information:

• pca: The result from the fast.svd() call.
• plot: The ggplot2 pca plot.
• table: The metadata used to make the pca plot.
• res: A table of the residual variance after each component by condition/batch.
• variance: A numeric list of the %variance remaining after each PC.

With that in mind, lets perform some more pca plots after normalizing the data and see how different they look.

normbatch_expt <- sm(normalize_expt(fission_expt, transform="log2", norm="quant", convert="cpm", batch="sva"))
fis_normbatchpca <- plot_pca(normbatch_expt, title="Normalized PCA with batch effect correction.")
fis_normbatchpca$plot

## ok, that caused the 0, 60, 15, and 30 minute samples to cluster nicely
## the 120 and 180 minute samples are still a bit tight

## pca_information provides some more information about the call to
## fast.svd that went into making the pca plot
fis_info <- pca_information(norm_expt, expt_factors=c("condition","batch","strain","minute"), num_components=6)

## More shallow curves in these plots suggest more genes in this principle component.

## The r^2 table shows that quite a lot of the variance in the data is explained by condition
head(fis_info$rsquared_table)

##   prop_var cumulative_prop_var condition  batch strain minute
## 1   36.642               36.64     99.14  0.017  0.123 98.832
## 2   14.010               50.65     96.67  0.382  0.006 95.833
## 3    8.432               59.08     66.40 25.100  1.834 61.469
## 4    5.807               64.89     38.72 26.589  6.086 29.265
## 5    5.179               70.07     88.88  5.137  0.316 87.439
## 6    3.633               73.70     15.16  0.950  7.738  4.997

## We can look at the correlation between the principle components and the factors in the experiment
## in this case looking at condition/batch vs the first 4 components.
fis_info$pca_cor

##               PC1       PC2      PC3     PC4      PC5     PC6
## condition 0.31602 -0.269379 -0.05825 -0.3246  0.03743 0.28533
## batch     0.01233 -0.060604  0.50079 -0.5105 -0.22602 0.09691
## strain    0.03507 -0.007689  0.13542 -0.2467  0.05617 0.27817
## minute    0.57717 -0.530995 -0.35561 -0.2228 -0.02303 0.08811

## And p-values to lend some credence(or not to those assertions)
fis_info$anova_p

##                 PC1       PC2      PC3      PC4    PC5     PC6
## condition 0.0604265 0.1121139 0.735783 0.053393 0.8284 0.09163
## batch     0.9430865 0.7255040 0.001865 0.001468 0.1850 0.57394
## strain    0.8390728 0.9645021 0.431012 0.146923 0.7449 0.10044
## minute    0.0002285 0.0008625 0.033295 0.191462 0.8940 0.60937

## Try again with batch removed data
batchnorm_expt <- sm(normalize_expt(fission_expt, batch="limma", norm="quant", transform="log2", convert="cpm"))
fis_batchnormpca <- plot_pca(batchnorm_expt, plot_title="limma corrected pca")
fis_batchnormpca$plot

test_pca <- pca_information(batchnorm_expt, expt_factors=c("condition","batch","strain","minute"), num_components=6)

## More shallow curves in these plots suggest more genes in this principle component.

Interesting, the batch normalized pca plot looks much the same as the normalized. The variances are in fact pretty much the exact same…

1.4 Look at the data distributions

We have some tools which provide visualizations of the distribution of the data:

fission_boxplot <- sm(plot_boxplot(fission_expt))
fission_boxplot

sf_expt <- sm(normalize_expt(fission_expt, norm="sf"))
fission_boxplot <- sm(plot_boxplot(sf_expt))
fission_boxplot

tm_expt <- sm(normalize_expt(fission_expt, norm="tmm"))
fission_boxplot <- sm(plot_boxplot(tm_expt))
fission_boxplot

rle_expt <- sm(normalize_expt(fission_expt, norm="rle"))
fission_boxplot <- sm(plot_boxplot(rle_expt))
fission_boxplot

up_expt <- sm(normalize_expt(fission_expt, norm="upperquartile"))
fission_boxplot <- sm(plot_boxplot(up_expt))
fission_boxplot

fission_density <- plot_density(norm_expt)
fission_density

fission_density <- plot_density(sf_expt)

## This data will benefit from being displayed on the log scale.

## If this is not desired, set scale='raw'

## Some entries are 0.  We are on log scale, setting them to 0.5.

## Changed 24130 zero count features.

fission_density

fission_density <- plot_density(tm_expt)

## This data will benefit from being displayed on the log scale.

## If this is not desired, set scale='raw'

## Some entries are 0.  We are on log scale, setting them to 0.5.

## Changed 24130 zero count features.

fission_density

compare_12 <- plot_qq_plot(fission_expt, x=1, y=2)

## Error in plot_qq_plot(fission_expt, x = 1, y = 2): could not find function "plot_qq_plot"

compare_12$log

## Error in eval(expr, envir, enclos): object 'compare_12' not found

1.5 See how they cluster

Ok, so we can further check out how the data cluster with respect to one another…

fission_cor <- plot_corheat(norm_expt)

fission_cor$plot
fission_cor <- plot_corheat(batchnorm_expt)

fission_cor$plot
fission_dis <- plot_disheat(norm_expt)

fission_dis$plot
fission_dis <- plot_disheat(batchnorm_expt)

fission_dis$plot

2 variancePartition

A new toy I have been introduced to is variancePartition, let us try that with this data.

test_varpart <- varpart(fission_expt, predictor=NULL, factors=c("condition","batch"))

## Attempting mixed linear model with: ~  (1|condition) + (1|batch)

## Fitting the expressionset to the model, this is slow.

## (Eg. Take the projected run time and mulitply by 3-6 and round up.)

## Projected run time: ~ 11 min

## Placing factor: condition at the beginning of the model.

test_varpart$percent_plot

test_varpart$partition_plot

## Here, let us test the variance contributed by strain, time, and replicate.
test_varpart <- varpart(fission_expt, predictor=NULL, factors=c("condition", "strain", "minute", "replicate"))

## Attempting mixed linear model with: ~  (1|condition) + (1|strain) + (1|minute) + (1|replicate)

## Fitting the expressionset to the model, this is slow.

## (Eg. Take the projected run time and mulitply by 3-6 and round up.)

## Projected run time: ~ 2 min

## Placing factor: condition at the beginning of the model.

test_varpart$percent_plot

test_varpart$partition_plot

YAY!

index.html

pander::pander(sessionInfo())

R version 3.4.1 (2017-06-30)

**Platform:** x86_64-pc-linux-gnu (64-bit)

locale: LC_CTYPE=en_US.utf8, LC_NUMERIC=C, LC_TIME=en_US.utf8, LC_COLLATE=en_US.utf8, LC_MONETARY=en_US.utf8, LC_MESSAGES=en_US.utf8, LC_PAPER=en_US.utf8, LC_NAME=C, LC_ADDRESS=C, LC_TELEPHONE=C, LC_MEASUREMENT=en_US.utf8 and LC_IDENTIFICATION=C

attached base packages: parallel, stats4, methods, stats, graphics, grDevices, utils, datasets and base

other attached packages: fission(v.0.110.0), SummarizedExperiment(v.1.6.3), DelayedArray(v.0.2.7), matrixStats(v.0.52.2), Biobase(v.2.36.2), GenomicRanges(v.1.28.4), GenomeInfoDb(v.1.12.2), IRanges(v.2.10.2), S4Vectors(v.0.14.3), BiocGenerics(v.0.22.0), ruv(v.0.9.6) and hpgltools(v.2017.01)

loaded via a namespace (and not attached): minqa(v.1.2.4), Rtsne(v.0.13), colorspace(v.1.3-2), colorRamps(v.2.3), rprojroot(v.1.2), htmlTable(v.1.9), corpcor(v.1.6.9), XVector(v.0.16.0), base64enc(v.0.1-3), ggrepel(v.0.6.5), bit64(v.0.9-7), AnnotationDbi(v.1.38.2), codetools(v.0.2-15), splines(v.3.4.1), doParallel(v.1.0.10), DESeq(v.1.28.0), robustbase(v.0.92-7), geneplotter(v.1.54.0), knitr(v.1.16), ade4(v.1.7-6), Formula(v.1.2-2), nloptr(v.1.0.4), pbkrtest(v.0.4-7), annotate(v.1.54.0), cluster(v.2.0.6), compiler(v.3.4.1), backports(v.1.1.0), Matrix(v.1.2-10), lazyeval(v.0.2.0), limma(v.3.32.4), acepack(v.1.4.1), htmltools(v.0.3.6), tools(v.3.4.1), gtable(v.0.2.0), GenomeInfoDbData(v.0.99.0), reshape2(v.1.4.2), Rcpp(v.0.12.12), gdata(v.2.18.0), preprocessCore(v.1.38.1), nlme(v.3.1-131), iterators(v.1.0.8), stringr(v.1.2.0), openxlsx(v.4.0.17), testthat(v.1.0.2), lme4(v.1.1-13), gtools(v.3.5.0), devtools(v.1.13.2), XML(v.3.98-1.9), edgeR(v.3.18.1), DEoptimR(v.1.0-8), MASS(v.7.3-47), zlibbioc(v.1.22.0), scales(v.0.4.1), BiocInstaller(v.1.26.0), RColorBrewer(v.1.1-2), yaml(v.2.1.14), memoise(v.1.1.0), gridExtra(v.2.2.1), ggplot2(v.2.2.1), pander(v.0.6.0), rpart(v.4.1-11), latticeExtra(v.0.6-28), stringi(v.1.1.5), RSQLite(v.2.0), highr(v.0.6), genefilter(v.1.58.1), foreach(v.1.4.3), RMySQL(v.0.10.12), checkmate(v.1.8.3), caTools(v.1.17.1), BiocParallel(v.1.10.1), rlang(v.0.1.1), bitops(v.1.0-6), evaluate(v.0.10.1), lattice(v.0.20-35), htmlwidgets(v.0.9), labeling(v.0.3), bit(v.1.1-12), plyr(v.1.8.4), magrittr(v.1.5), variancePartition(v.1.6.0), DESeq2(v.1.16.1), R6(v.2.2.2), gplots(v.3.0.1), Hmisc(v.4.0-3), DBI(v.0.7), foreign(v.0.8-69), withr(v.1.0.2), mgcv(v.1.8-17), survival(v.2.41-3), RCurl(v.1.95-4.8), nnet(v.7.3-12), tibble(v.1.3.3), crayon(v.1.3.2), KernSmooth(v.2.23-15), rmarkdown(v.1.6), locfit(v.1.5-9.1), grid(v.3.4.1), sva(v.3.24.4), data.table(v.1.10.4), blob(v.1.1.0), digest(v.0.6.12), xtable(v.1.8-2), genoPlotR(v.0.8.6) and munsell(v.0.4.3)